{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "tfidf_test.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/henrygas/unsupervised_learning/blob/master/tfidf_test.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DDe6UpO7AKi2",
        "colab_type": "text"
      },
      "source": [
        "## 1. 装载Google云盘"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M6qyUKirAFWP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 127
        },
        "outputId": "27b193d3-8a76-46d0-d1b0-c1b64a577bee"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount(\"/content/drive\")"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G8Ul2nX6ApMQ",
        "colab_type": "text"
      },
      "source": [
        "## 2. 定位当前工作目录"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_AVIcGZpAs4Q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# !ls drive/My\\ Drive/app/tfidf_test\n",
        "import os\n",
        "os.chdir(\"./drive/My Drive/app/tfidf_test\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IAXiXlW6BNoa",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "802f7df8-f0f0-4e3c-e435-fcb6189b7578"
      },
      "source": [
        "!ls"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "data  out\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Edl8nwbBbgN",
        "colab_type": "text"
      },
      "source": [
        "## 3. 学习词袋模型CountVectorizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yxe6F7RrBm55",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 323
        },
        "outputId": "60c416e7-d44c-4c0b-ffb5-0341d07a2362"
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "vec = CountVectorizer()\n",
        "vec.get_params()"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'analyzer': 'word',\n",
              " 'binary': False,\n",
              " 'decode_error': 'strict',\n",
              " 'dtype': numpy.int64,\n",
              " 'encoding': 'utf-8',\n",
              " 'input': 'content',\n",
              " 'lowercase': True,\n",
              " 'max_df': 1.0,\n",
              " 'max_features': None,\n",
              " 'min_df': 1,\n",
              " 'ngram_range': (1, 1),\n",
              " 'preprocessor': None,\n",
              " 'stop_words': None,\n",
              " 'strip_accents': None,\n",
              " 'token_pattern': '(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
              " 'tokenizer': None,\n",
              " 'vocabulary': None}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qvxBkYDzB1m1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "5d0d25e2-018b-4151-8c37-4b7f23384ca2"
      },
      "source": [
        "corpos = [\n",
        "          \"This is the first document.\",\n",
        "          \"This is the second second document.\",\n",
        "          \"And the third one.\",\n",
        "          \"Is this the first document?\"\n",
        "]\n",
        "X = vec.fit_transform(corpos)\n",
        "X"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<4x9 sparse matrix of type '<class 'numpy.int64'>'\n",
              "\twith 19 stored elements in Compressed Sparse Row format>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zeSbkcNUCehT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "0c1e56d4-1d32-4fb5-be7c-3ac9efde62ba"
      },
      "source": [
        "vec.get_feature_names()"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9_3QFLHsCqfy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "outputId": "53fe8162-cb6c-4f83-9499-60d13fa2717a"
      },
      "source": [
        "X.toarray()"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0, 1, 1, 1, 0, 0, 1, 0, 1],\n",
              "       [0, 1, 0, 1, 0, 2, 1, 0, 1],\n",
              "       [1, 0, 0, 0, 1, 0, 1, 1, 0],\n",
              "       [0, 1, 1, 1, 0, 0, 1, 0, 1]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xoUney05GPT9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 179
        },
        "outputId": "20d168f7-d277-4526-8fcd-b6c567871994"
      },
      "source": [
        "# 词汇表\n",
        "vec.vocabulary_"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'and': 0,\n",
              " 'document': 1,\n",
              " 'first': 2,\n",
              " 'is': 3,\n",
              " 'one': 4,\n",
              " 'second': 5,\n",
              " 'the': 6,\n",
              " 'third': 7,\n",
              " 'this': 8}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J8jVUUUwGiC4",
        "colab_type": "text"
      },
      "source": [
        "针对其他文本进行词袋处理时，可以直接使用现有的词汇表"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8hKvfRS3Gorp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "vec_new = CountVectorizer(vocabulary=vec.vocabulary_)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "45YpV88lG4zv",
        "colab_type": "text"
      },
      "source": [
        "在将来的fit_transform()方法中，在词汇表vocabulary_中未出现的词将被完全忽略。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R8HY7ccyHLTY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "2221cdd4-3b91-4eb7-b5c3-2cf9d3910f70"
      },
      "source": [
        "vec_new.fit_transform([\"Something third new.\"]).toarray()"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0, 0, 0, 0, 0, 0, 0, 1, 0]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u1xIG1qpIHRG",
        "colab_type": "text"
      },
      "source": [
        "第二个例子，增加出现文档频率的要求"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LHBnlZpSIMJb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "30960a94-09e8-434f-e37f-77d8d69ac8b3"
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "import numpy as np\n",
        "\n",
        "vec_df = CountVectorizer(min_df=3, ngram_range=(1, 1))\n",
        "content=np.array([\n",
        "    '<s[NULL]cript>alert(1)</s[NULL]cript>X</a>',\n",
        "    '\\'><video><source o?UTF-8?Q?n?error=\"alert(1)\">',\n",
        "    '\\'><FRAMESET><FRAME RC=\"\"+\"javascript:alert(\\'X\\');\"></FRAMESET>',\n",
        "    '\"></script>\\'//<svg \"%0Aonload=alert(1) //>',\n",
        "    '\"></script><img \\'//\"%0Aonerror=alert(1)// src>',\n",
        "    'id%3Den%22%3E%3Cscript%3Ealert%28%22AKINCILAR%22%29%3C/script%3E',\n",
        "    '?a%5B%5D%3D%22%3E%3Cscript%3Ealert%28document.cookie%29%3C/script%3E',\n",
        "    '><iframe src=\"data:data:javascript:,% 3 c script % 3 e confirm(1) % 3 c/script %3 e\">',\n",
        "    '?mess%3D%3Cscript%3Ealert%28document.cookie%29%3C/script%3E%26back%3Dsettings1',\n",
        "    'title%3D%3Cscript%3Ealert%28%22The%20Best%20XSSer%22%29%3C/script%3E',\n",
        "    '<script charset>alert(1);</script charset>',\n",
        "    '\"><meta charset=\"x-mac-farsi\">??script ??alert(1)//??/script ??',\n",
        "    '</script><script>/*\"/*\\'/**/;alert(1)//</script>#',\n",
        "])\n",
        "\n",
        "X_2 = vec_df.fit_transform(content)\n",
        "vec_df.get_feature_names()"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['22', '29', '3c', '3cscript', '3d', '3e', '3ealert', 'alert', 'script']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sN4O5EHnJQ8p",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "446b96b7-f1cc-42a6-c1d0-532509642a8e"
      },
      "source": [
        "X_2"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<13x9 sparse matrix of type '<class 'numpy.int64'>'\n",
              "\twith 44 stored elements in Compressed Sparse Row format>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zhItJO_wJShu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 251
        },
        "outputId": "0efd4d28-d97d-4cc5-8062-3a3751078628"
      },
      "source": [
        "X_2.toarray()"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0, 0, 0, 0, 0, 0, 0, 1, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 1, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 1, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 1, 1],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 1, 1],\n",
              "       [2, 1, 1, 1, 0, 2, 1, 0, 1],\n",
              "       [1, 1, 1, 1, 1, 2, 1, 0, 1],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 2],\n",
              "       [0, 1, 1, 1, 1, 1, 1, 0, 1],\n",
              "       [1, 1, 1, 1, 1, 1, 1, 0, 1],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 1, 2],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 1, 2],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 1, 3]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UAoDHhTbJmrC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 179
        },
        "outputId": "ab640474-5290-44c7-eee5-44afdef39bc7"
      },
      "source": [
        "vec_df.vocabulary_"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'22': 0,\n",
              " '29': 1,\n",
              " '3c': 2,\n",
              " '3cscript': 3,\n",
              " '3d': 4,\n",
              " '3e': 5,\n",
              " '3ealert': 6,\n",
              " 'alert': 7,\n",
              " 'script': 8}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kr7dC77LJ3A7",
        "colab_type": "text"
      },
      "source": [
        "## 4. 学习TF-IDF"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GCtoktpxKIWT",
        "colab_type": "text"
      },
      "source": [
        "是一种加权技术，是词频(TF)和逆文档频率(IDF)的乘积\n",
        "+ 在一个文档中高频出现，但在其他文档中很少出现的词，最能反映当前文档的特色，于是这样的词，就被施以高权重；\n",
        "+ 在所有文档中都高频出现，意味着TF高，但IDF低，这样的词，大多属于无意义的常用词，就被施以低权重；\n",
        "+ 从上面的分析可以看出，文档数越多，IDF的影响越大，结果也越公正；如果只有一篇文档，那么TF就和TF-IDF相同了，这样就不能看出高频出现的词，到底是有意义的专业词，还是无意义的常用词了。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fLjjJjpgO0E2",
        "colab_type": "text"
      },
      "source": [
        "简单理解: 一个人，假设他对所有人都很好，那他对你的好，就不那么醒目了；如果他对其他所有人都很苛刻，但对你还不错，那么你就知道，他是真的对你好了"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h_pMORKzO2BL",
        "colab_type": "text"
      },
      "source": [
        "但这个模型，有以下三个缺陷，下面举例说明:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r8TBab7SPFn6",
        "colab_type": "text"
      },
      "source": [
        "(1) 第一个缺陷，比如一个人，他可能对水瓶座的人很有好感，于是他对遇到的所有水瓶，都很好，那如果你参考的样本里，恰好大部分都是水瓶，那他对你的好，就被忽略了，因为不够醒目嘛，但实际上，喜欢水瓶恰恰是他的一个性质"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UUyRjP3dPLTv",
        "colab_type": "text"
      },
      "source": [
        "(2) 第二个缺陷，就是没考虑这个人当时的处境；因为你观察到了结果(喜欢/不喜欢)，而忽略了他产生这个结果时的实时处境，所以你也做出了误判"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NZ_sbzKlPO2L",
        "colab_type": "text"
      },
      "source": [
        "(3) 第三个缺陷，他可能只是恰好对你的某个特质感兴趣，并不是真的喜欢你这个人，但你通过观测结果，也无法分析出这个结论"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jA0PtppAPSUC",
        "colab_type": "text"
      },
      "source": [
        "这就是TF-IDF的三大缺陷：\n",
        "+ 没有考虑到特征项在类间和类内的分布情况;\n",
        "+ 没有考虑特征项的位置因素对文本的区分度;\n",
        "+ 会对一些无意义的生僻字施以高权重，但并不能提高对文本的区分度"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pcr0QglyJ6da",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "content = [\n",
        "    \"I went to your house and steal your rice to eat, but bitten by your dog, so I think you need to pay me.\",\n",
        "    \"I thought to play with you this afternoon, however it will rain later, so you have to play with yourself.\",\n",
        "    \"I decided to make friends with you when I saw your wife first sight.\"\n",
        "]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Plsu1wwwUoIw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "tfidf_vec = TfidfVectorizer()\n",
        "X_tfidf = tfidf_vec.fit_transform(content).toarray()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WJOlaL4jU5vn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "2c436a92-3ab8-4b02-8544-5f731a071b99"
      },
      "source": [
        "X_tfidf.shape"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(3, 38)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "03yeNf2xWPni",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 449
        },
        "outputId": "20a68404-3707-4702-c1ec-945dffb37d0b"
      },
      "source": [
        "X_tfidf"
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.        , 0.20729109, 0.20729109, 0.20729109, 0.20729109,\n",
              "        0.        , 0.20729109, 0.20729109, 0.        , 0.        ,\n",
              "        0.        , 0.20729109, 0.        , 0.        , 0.        ,\n",
              "        0.        , 0.20729109, 0.20729109, 0.20729109, 0.        ,\n",
              "        0.        , 0.20729109, 0.        , 0.        , 0.15765022,\n",
              "        0.20729109, 0.20729109, 0.        , 0.        , 0.36728838,\n",
              "        0.20729109, 0.        , 0.        , 0.        , 0.        ,\n",
              "        0.12242946, 0.47295066, 0.        ],\n",
              "       [0.22540243, 0.        , 0.        , 0.        , 0.        ,\n",
              "        0.        , 0.        , 0.        , 0.        , 0.        ,\n",
              "        0.22540243, 0.        , 0.22540243, 0.22540243, 0.22540243,\n",
              "        0.        , 0.        , 0.        , 0.        , 0.45080485,\n",
              "        0.22540243, 0.        , 0.        , 0.        , 0.17142435,\n",
              "        0.        , 0.        , 0.22540243, 0.22540243, 0.26625261,\n",
              "        0.        , 0.        , 0.        , 0.22540243, 0.34284871,\n",
              "        0.26625261, 0.        , 0.22540243],\n",
              "       [0.        , 0.        , 0.        , 0.        , 0.        ,\n",
              "        0.31855448, 0.        , 0.        , 0.31855448, 0.31855448,\n",
              "        0.        , 0.        , 0.        , 0.        , 0.        ,\n",
              "        0.31855448, 0.        , 0.        , 0.        , 0.        ,\n",
              "        0.        , 0.        , 0.31855448, 0.31855448, 0.        ,\n",
              "        0.        , 0.        , 0.        , 0.        , 0.18814341,\n",
              "        0.        , 0.31855448, 0.31855448, 0.        , 0.2422689 ,\n",
              "        0.18814341, 0.2422689 , 0.        ]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TsE7QV7jWVha",
        "colab_type": "text"
      },
      "source": [
        "## 5. TF-IDF和词袋的配合使用"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iFXjCb95WbyL",
        "colab_type": "text"
      },
      "source": [
        "前面我们看出，TF-IDF的作用是找出某些词用以提取该词所在文档的特色，而词袋模型的作用是找出文档中出现频率高的词；\n",
        "\n",
        "而有特色的词，既有可能是有意义的专业词，又有可能是无意义的生僻词，可以考虑将两者结合起来，具体步骤是:\n",
        "\n",
        "+ 通过词袋模型挑出一堆热词;\n",
        "+ 再用TF-IDF对这些热词加权，得到有区分度的词。\n",
        "\n",
        "\n",
        "这样就可以通过控制热词数，来将无意义的常用词和有意义的专业词都框进来；\n",
        "\n",
        "然后再通过加权技术，排除掉无意义的常用词，争取留下有意义的专业词。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LnrKfcLzUlqA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.feature_extraction.text import TfidfTransformer\n",
        "\n",
        "content=np.array([\n",
        "    '<s[NULL]cript>alert(1)</s[NULL]cript>X</a>',\n",
        "    '\\'><video><source o?UTF-8?Q?n?error=\"alert(1)\">',\n",
        "    '\\'><FRAMESET><FRAME RC=\"\" \"javascript:alert(\\'X\\');\"></FRAMESET>',\n",
        "    '\"></script>\\'//<svg \"onload=alert(1) //>',\n",
        "    '\"></script><img \\'//\"onerror=alert(1)// src>',\n",
        "    'id=en\"><script>alert(\"AKINCILAR\")</script>',\n",
        "    '?a[]=\"><script>alert(document.cookie)</script>',\n",
        "    '><iframe src=\"data:data:javascript:,% 3 c script % 3 e confirm(1) % 3 c/script %3 e\">',\n",
        "    '?mess=<script>alert(document.cookie)</script>&back=settings1',\n",
        "    'title=<script>alert(\"The Best XSSer\")</script>',\n",
        "    '<script charset>alert(1);</script charset>',\n",
        "    '\"><meta charset=\"x-mac-farsi\">??script ??alert(1)//??/script ??',\n",
        "    '</script><script>/*\"/*\\'/**/;alert(1)//</script>#',\n",
        "])\n",
        "\n",
        "count_vec = CountVectorizer()\n",
        "X_count = count_vec.fit_transform(content)\n",
        "\n",
        "tfidf_transform = TfidfTransformer()\n",
        "X_tfidf = tfidf_transform.fit_transform(X_count)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e1wOLp2LYRYZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 472
        },
        "outputId": "68509b2a-e6d2-4a90-e19c-ce16267fe5dc"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "df = pd.DataFrame(data=X_tfidf.toarray(), columns=count_vec.get_feature_names())\n",
        "df"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>akincilar</th>\n",
              "      <th>alert</th>\n",
              "      <th>back</th>\n",
              "      <th>best</th>\n",
              "      <th>charset</th>\n",
              "      <th>confirm</th>\n",
              "      <th>cookie</th>\n",
              "      <th>cript</th>\n",
              "      <th>data</th>\n",
              "      <th>document</th>\n",
              "      <th>en</th>\n",
              "      <th>error</th>\n",
              "      <th>farsi</th>\n",
              "      <th>frame</th>\n",
              "      <th>frameset</th>\n",
              "      <th>id</th>\n",
              "      <th>iframe</th>\n",
              "      <th>img</th>\n",
              "      <th>javascript</th>\n",
              "      <th>mac</th>\n",
              "      <th>mess</th>\n",
              "      <th>meta</th>\n",
              "      <th>null</th>\n",
              "      <th>onerror</th>\n",
              "      <th>onload</th>\n",
              "      <th>rc</th>\n",
              "      <th>script</th>\n",
              "      <th>settings1</th>\n",
              "      <th>source</th>\n",
              "      <th>src</th>\n",
              "      <th>svg</th>\n",
              "      <th>the</th>\n",
              "      <th>title</th>\n",
              "      <th>utf</th>\n",
              "      <th>video</th>\n",
              "      <th>xsser</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.127851</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.701304</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.701304</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.179349</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.491893</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.491893</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.491893</td>\n",
              "      <td>0.491893</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.139040</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.38134</td>\n",
              "      <td>0.762681</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.328854</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.38134</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.239872</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.657888</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.277179</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.657888</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.208634</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.572212</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.572212</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.241083</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.493455</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>0.510113</td>\n",
              "      <td>0.185992</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.510113</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.510113</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.429839</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.238848</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.564915</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.564915</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.551991</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.349271</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.698542</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.349271</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.301199</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.294308</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.301199</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.157925</td>\n",
              "      <td>0.433135</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.373520</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.373520</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.433135</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.364975</td>\n",
              "      <td>0.433135</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.165681</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.454406</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.382898</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.454406</td>\n",
              "      <td>0.454406</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.454406</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.186608</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.882718</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.431261</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.170248</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.402664</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.466931</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.466931</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.466931</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.393452</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.277167</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.960822</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    akincilar     alert      back  ...       utf     video     xsser\n",
              "0    0.000000  0.127851  0.000000  ...  0.000000  0.000000  0.000000\n",
              "1    0.000000  0.179349  0.000000  ...  0.491893  0.491893  0.000000\n",
              "2    0.000000  0.139040  0.000000  ...  0.000000  0.000000  0.000000\n",
              "3    0.000000  0.239872  0.000000  ...  0.000000  0.000000  0.000000\n",
              "4    0.000000  0.208634  0.000000  ...  0.000000  0.000000  0.000000\n",
              "5    0.510113  0.185992  0.000000  ...  0.000000  0.000000  0.000000\n",
              "6    0.000000  0.238848  0.000000  ...  0.000000  0.000000  0.000000\n",
              "7    0.000000  0.000000  0.000000  ...  0.000000  0.000000  0.000000\n",
              "8    0.000000  0.157925  0.433135  ...  0.000000  0.000000  0.000000\n",
              "9    0.000000  0.165681  0.000000  ...  0.000000  0.000000  0.454406\n",
              "10   0.000000  0.186608  0.000000  ...  0.000000  0.000000  0.000000\n",
              "11   0.000000  0.170248  0.000000  ...  0.000000  0.000000  0.000000\n",
              "12   0.000000  0.277167  0.000000  ...  0.000000  0.000000  0.000000\n",
              "\n",
              "[13 rows x 36 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EeFXWW0fZbhd",
        "colab_type": "text"
      },
      "source": [
        "tf-idf值越高区分语句的效果越好；\n",
        "\n",
        "但tf-idf值越高，泛化能力越差；\n",
        "\n",
        "选择特征向量时，应该选择泛化能力强，即tf-idf值较低的。\n",
        "\n",
        "在这个例子里：\n",
        "+ alert的权重普遍比较低，但它的泛化能力较好；\n",
        "+ script的权重也比较低，有一定的泛化能力，但相比alert还是略差一点。\n",
        "+ 其他的词，往往在个别的单个文档中出现，权值虽高，但不具有泛化能力，不应该作为特征向量。"
      ]
    }
  ]
}